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Positive Behavioral Interventions and Supports (PBIS) is an evidence-based approach to addressing behavior issues in 
schools. Typically, the Minnesota Department of Education and the Regional Training Projects sponsor three training 
sessions per year for schools that are participating in Minnesota’s two-year training sequence for PBIS. However, due 
to the ongoing COVID-19 global pandemic, the Minnesota State Leadership Team (SLT) modified PBIS training to 
align with individual school changes in learning plans (1.e. distance learning, hybrid, or in-person). 


This report summarizes training survey results for both of these types of training and their implementation fidelity 
measurement for Winter 2020-21. School teams are comprised of school 

staff members and administrators. Participating school teams complete one More information about PBIS in Minnesota 
survey per team at each training. This report summarizes the results for: can be found at: www.pbismn.org. 


=" 70 school teams that are in their second year of training (Cohort 15) 
=" 53 school teams are in their first year of training (Cohort 16) 


Training attendance 


In non-pandemic times, schools in their first year of training (Cohort 16 in 2020-22) receive two days of training at 
each of three points during the year, and schools in their second year of training (Cohort 15 in 2019-21) receive one day 
at each of three points during the year. For Winter 2020-2021 training, school teams could choose between either of 
these PBIS training delivery methods: 


=" Live training: school teams could attend a live training using a virtual platform over the course of one or two 
days, depending on which cohort their school is a part of, or, 

"Asynchronous training: school teams watched PBIS training videos, met virtually with coaches, and 
completed other various tasks over the course of six weeks (though some schools needed more time). 


Training surveys were administered based on type of training method selected by school teams. As this differs from 
Fall 2020 and previous school years where training was held in-person or only via a live, virtual platform, this shift 
should be considered when interpreting results and comparing data across years. School teams are required to attend 
training. Staff and trainers at the Regional Implementation Projects (RIPs) keep track of team attendance and 
administrator attendance at training, however this was difficult to do this year as all training was administered 
online. Administrator attendance is recommended, and their attendance is monitored as a potential indicator of buy- 
in to the school’s PBIS initiative. 


A total of 30 school teams from both cohorts completed a survey for the live training and 20 school teams completed an 
end-point training survey for the asynchronous training. Note that there were two surveys for the asynchronous training: 
one survey at the midpoint and another one at the end. After the RIPs received feedback from school teams that they 
were overwhelmed with the number of surveys they needed to complete and low participation in the mid-point 
asynchronous survey, the midpoint survey was closed and schools were no longer encouraged to complete it. 


Figure | below illustrates the number of school teams who completed training surveys based on region, cohort, and 
training method. Six school teams from Cohort 15 and one school team from Cohort 16 completed both a mid-point 


and end-point survey for the asynchronous training; these are included in the number of responses for each type of 
training survey. The overall response rate for the winter training surveys was very low compared with other years. 
Given the ongoing pandemic and the shift in training delivery methods, this is not unexpected. Due to this low 
participation rate, training survey results are not disaggregated by region. 


1. Winter 2020-2021 survey responses from school teams by cohort and region (N=number of schools in training) 


Cohort 15 Cohort 16 


Metro North* South Metro North South 
(N=16) | (N=38) | (N=15) | (N=22)  (N=16)  (N=14) 


Live training 3 4 8 7 5 3 
Asynchronous training--Midpoint ri 4 2 3 1 
Asynchronous training—end point 2 6 5 - 3 4 


Overall satisfaction with training 
School teams were asked to indicate how useful the training was in helping their team implement PBIS at their school 
sites. Key findings include: 


» Most school teams that completed the survey said winter 2020 training was very useful or somewhat useful 
(Figure 2). 


» A few school teams in Cohort 15 indicated winter 2020 training was not very useful. 


2. School team ratings: “Overall, how useful was this PBIS training in terms of helping your team to implement PBIS in your school?” 


Cohort 15 Cohort 16 

End-point End-point 
Live Asynchronous Live Asynchronous 
(N=15) (N=12) (N=15) (N=7) 
Very useful 10 1 5 4 
Somewhat useful 4 g 10 3 
Not very useful 1 2 = -- 
Not at all useful fe = 


Note. This question was not asked on the mid-point survey. 
Action planning 


Cohort 15 and 16 school teams that participated in either type of training format were asked to list three action planning 
items their school has been working on the most since Fall 2020 training. Cohort 16 school teams that completed the 
asynchronous training were not asked about their action planning items. Key findings include: 


» The most common action planning items listed by Cohort 15 school teams included acknowledgements and 
recognition, data, and updating their PBIS matrix (Figure 3). 
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» Half of school teams from Cohort 16 that participated in the live training reported they are working on major and 
minor behaviors as part of their PBIS action plan (Figure 4). 


3. Action planning items identified by school teams—Cohort 15 


Cohort 15 
End-point 
Live Asynchronous 
(N=15) (N=12) 

Acknowledgements/Recognition 6 8 
Buy-in (staff & student motivation) 5 

Data/using data 4 8 
Expectations (teaching/student) 5 - 
Family/community involvement 5 3 
Matrix 6 5 


Note: Other less commonly mentioned action planning items include: accommodations for distance leaming, artifacts, professional development, 
SEL/trauma informed PBIS, training school staff, and tier 2 and 3 interventions. 


4. Top action planning items identified by school teams—Cohort 16 


Cohort 16 
Live 
(N=14) 
Behavior/Majors & Minors i 
Family & community involvement 6 
Flowchart 4 
Matrix 4 


Notes: Cohort 16 school teams that participated in the asynchronous training were not asked about their action planning items. 
Other less commonly mentioned action planning items include: feedback and acknowledgement, and using data. 


Most useful training components 


School teams from both cohorts were asked in the open-ended format what parts of the live and asynchronous trainings 
were the most useful for their PBIS teams. 


School teams from Cohort 16 that participated in either the live training or the asynchronous training said that work 
time (n=11), coaching and supportive facilitators (n=9), and examples of key PBIS components (n=7) were the 
most useful parts of training. Below are some of their comments, edited for clarity: 


Work time as a team was incredibly helpful. Our external coach, was very helpful at guiding us--she made statements 
that helped clarify our thinking. She helped us with our timeline. She was very supportive. 


We liked all the PowerPoint slides that we can use or go back to as a resource. The website is also amazing for our 
coach to access our materials. All the tangible resources were fantastic and so helpful. Having our coach in our group 
chat was also great for clarification and resource purposes. 


We benefited from hearing more about different acknowledgment systems as well as different forms for behavior referrals. 
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Similarly, most school teams from Cohort 15 stated that work time (n=13) was the most useful part about training. 
Three teams that participated in the live training noted that they appreciated the breakout sessions. Team members 
from two schools that participated in the asynchronous training noted that they appreciated the ideas and engagement 
from other schools. Two other school teams that participated in the asynchronous training mentioned using and 
engaging with data. Below are some of their comments, edited for clarity: 


Work time and hearing from other schools that we are in the same boat with them about issues we are experiencing. 
In our breakout time with our team, we were able to discuss topics that we felt were important to for our school. 
The suggestions about incentives and engagement from other districts were helpful. 


The training prompted us to think more about our data. 


Areas for improvement 


School teams were also asked what they would improve about Winter 2020 training to help their PBIS teams. Six 
school teams in Cohort 15 that participated in the live training had suggestions around training format and timing, 
while three teams suggested that trainers provide more examples of how to implement PBIS in their school. 


We need examples of how to translate PBIS stuff to online and hybrid. 


We'd like suggestions for how to work around challenges with implementation at school. We’re running into a lot of 
roadblocks with how to actually put anything into action. 


Start training earlier and end it earlier. We need more time to view videos and complete worksheets. 


We'd like more time. Things seemed a little jam packed. It makes sense that they are fitting in two days in one day, but 
it almost seemed rushed. 


Four school teams in Cohort 15 and one school team in Cohort 16 that participated in the asynchronous training said 
they would prefer to hold in-person training, though some acknowledged that is not possible currently due to the 
pandemic. 


We'd prefer full day trainings like normal to get away from the school building and focus on PBIS implementation. We 
have to deal with a lot of distractions when we’re in the school building. 


It's too hard to be together with Covid, but we got more out of our in-person trainings. 


If we were able to meet in person. I feel that it would be very beneficial for our team to get to work with the facilitators 
in person. (I know that is not possible due to our worldwide pandemic right now). 


Six school teams that participated in either the live training or the asynchronous training in Cohort 16 said they were 
overwhelmed by the amount of content, action planning steps, and longer presentations. 

I feel like the amount of material that we got was overwhelming, it was hard to know where to start. We didn't know 
exactly what we should be working on at any given time. 


The full day is tough to sit through - there's so many action items as well, and it becomes overwhelming. There's no 
time to implement things before coming back and learning more. 


Five school teams in both cohorts that participated in either the live or asynchronous training model noted that their 
school was struggling with PBIS training due to the ongoing COVID-19 pandemic: 


It is just a hard year with Covid-19 and switching models. Our brains are at capacity and our staff are overwhelmed. 
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I cannot identify one thing that would be more beneficial, I believe the [asynchronous] format would have been more 
beneficial had we not already been trying to juggle different learning formats throughout the training period. At times, 
it felt like one extra responsibility. 


We had a difficult time getting our entire team together for trainings due to COVID-related issues. It was extremely 
helpful to have the online modules as an option. 


Ratings of training effectiveness 


School teams that attended the live training were asked how strongly they agree or disagree with eight statements 
about the effectiveness of the trainings. These statements included whether or not the training content was clearly 
presented, if the training enhanced their understanding of PBIS, if the training content will assist PBIS implementation 
in their school, and whether or not they are confident to share the topics they learned at training with other school staff. 
Schools were also asked about PBIS team members’ confidence in their ability to share implementation fidelity and 
outcome data with other school staff. Since PBIS training was moved to an online format, school teams who 
participated in the live training were asked about the extent they understood concepts in the online training format as 
well as technical difficulties. Some key findings include: 


» Most school teams from both cohorts that participated in the live training either strongly agreed or somewhat 
agreed that the training content was presented clearly, the training content will improve PBIS practices in their 
schools, and that they are confident in sharing topics learned at training with other school staff (Figure 5). Some 
school teams from Cohort 15 disagreed that the training enhanced their understanding of PBIS. 


» More school teams from Cohort 16 said they somewhat agreed that the training content was clearly presented 
and that they are confident in their ability to share what they learned at PBIS training with other school staff. 


» A few school teams from Cohort 16 disagreed they were confident to share either implementation fidelity or 
outcome data. Similarly, a few school teams from Cohort 16 also indicated they struggled with technical 
difficulties during training and they had trouble understanding PBIS training concepts in the online format. 


5. Ratings of training effectiveness: Live training 


Cohort 15 Cohort 16 


(N=14) (N=15) 
The training content was clearly presented. 
Strongly agree 8 5) 
Somewhat agree 6 9 
Somewhat disagree - 1 
Strongly disagree 
The training enhanced our team members’ understanding of PBIS. 
Strongly agree 4 8 
Somewhat agree 6 Uf 
Somewhat disagree 4 
Strongly disagree - 
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5. Ratings of training effectiveness: Live training (continued) 


Cohort 15 Cohort 16 
(N=14) (N=15) 


The training content will improve PBIS practices in our school. 
6 11 


Strongly agree 
Somewhat agree 8 4 
Somewhat disagree 7 - 
Strongly disagree - 


We are confident in our ability to share the topics we learned 
today with other school staff. 


Strongly agree | 9 5 
Somewhat agree 5) 9 
Somewhat disagree - 1 
Strongly disagree | = | 


We are confident in our ability to share implementation fidelity 
data (i.e., SAS, TFl) with other school staff. 


Strongly agree 10 3 
Somewhat agree 4 7 
Somewhat disagree - 5 
Strongly disagree - - 


We are confident in our ability to share outcome data (e.g., office 
discipline referrals, or ODR) with other school staff. 


Strongly agree 9 | 


5 
Somewhat agree 3 6 
Somewhat disagree 2 3 
Strongly disagree - 1 


We were able to understand the concepts and learning modules in 
the online training format. 


Strongly agree 9 9 
Somewhat agree 5 3 
Somewhat disagree - 3 
Strongly disagree - - 


Our team had no trouble/technical difficulties with the online 
training format. 


Strongly agree 10 7 7 
Somewhat agree 4 5 
Somewhat disagree 2 


Strongly disagree - 1 


School teams that completed the asynchronous training were also asked how strongly they agreed whether the training 
tasks and expectations were easy to understand, whether they were able to understand the concepts and training models 
in the asynchronous format, and whether or not their school experienced technical difficulties with the asynchronous 
format. Key findings include: 


» 4 


Most school teams from both cohorts that participated in the asynchronous training agreed training tasks and 
expectations were easy to understand, that they were able understand the concepts and learning models in the 
asynchronous format, and that they did not have any trouble or technical difficulties with the asynchronous 
format (Figure 6). However, more schools in Cohort 16 noted that they only “somewhat agreed” that they were 
able to understand the concepts and learning modules in the asynchronous training format. 


6. Ratings of training effectiveness: Asynchronous training 


Cohort 15 Cohort 16 
(N=12) (N=7) 


Our training tasks and expectations were easy to understand. 


Strongly agree ul 3 
Somewhat agree 5 3 
Somewhat disagree - 1 
Strongly disagree - 


We were able to understand the concepts and learning modules 
in the asynchronous training format. 


Strongly agree 6 2 
Somewhat agree 6 5 
Somewhat disagree = n 
Strongly disagree - 


Our team had no trouble/technical difficulties with the 
asynchronous format. 


Strongly agree 7 4 
Somewhat agree 5 3 
Somewhat disagree 

Strongly disagree - 


NOTE: Results above are from the end-point asynchronous training survey only. 
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Acknowledgements and feedback 


School teams from Cohort 15 and 16 that participated in the asynchronous training were asked if they were using 
acknowledgements this year for PBIS. If school teams said they were using acknowledgements this year, they were 
asked how acknowledgements are being delivered to students. Ten out of 12 schools from Cohort 15 and 3 out of 7 
schools from Cohort 16 said they are using acknowledgements this year. Of those that said they are using 
acknowledgements, most school teams from Cohort 15 (n=8) and Cohort 16 (n=3) said they are delivering these 
acknowledgements to students in both an online and virtual format. 


Tiered Fidelity Inventory 


The PBIS State Leadership Team expects that school The TFI measures implementation fidelity for each of 
teams in their first year of training (Cohort 16) will score the three tiers of PBIS. It is designed to be taken three 
at least 40% on their Tier 1 TF] at winter training. As it times per year while a school team is in training. In 

is their second year of training, Cohort 15 school teams Minnesota, schools complete a TFI at each PBIS 

are expected to meet the 70% implementation fidelity training. Some schools also completed a “walk-through 


benchmark at winter training. Some key findings from ee EWEN: eu gue eels fe eae 2 compeling 
. . the TFI facilitation session. Schools in training are 
the winter 2020 TFI assessments include: 


currently only taking the Tier 1 portion of the TFI 


= The overall average Tier 1 TFI scores for assessment. In order to be implementing PBIS with 
Cohorts 15 increased by 12 percentage points fidelity, a school needs to score 70 percent or better for 
since fall 2020, and the overall average for each of the three tiers of PBIS. 


Cohort 16 has increased by 16 percentage 
points since fall 2020 (Figure 7). 
=" Despite the increase in average scores from fall to winter 2020, the number of school teams that 
completed a TFI decreased. Sixty-six percent (46 out of 70) Cohort 15 school teams completed a TFI in 
winter 2020 compared with 80% (n=56) in fall 2020. For Cohort 16, 71% (38 out of 53) school teams 
completed a TFI during the same time period compared with 94% (n=50) of school teams that completed a 
TFI in Fall 2020. 
=" The average score for all teams was 66% for Cohort 15, which was below the 70% winter benchmark for 
schools in year two of training. On average, school teams in Cohort 15 from the South region are 
implementing PBIS with fidelity according to their TFI scores, while school teams from the North and Metro 
fell just below this threshold. 
"Cohort 16 school teams across the state had an average tier | TFI score of 41%. School teams from the North 
and South region, on average, exceeded the winter training benchmark (40%), while the Metro region was 
slightly below (Figure 8). 
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7. Average Tier 1 TFl scores by region—Cohort 15 (Fall 2019 through Winter 2020) 


68% 
66% 9 0 66% 
61% 707 som 0% aks 5B %xn 407, 

46% “47% ’ 

; 40% 
z pat i ‘q 
@ “ 
Metro North South Total average across all 
regions for all completed TFls 
= Fall 2019 = Winter 2019 = Spring/End of Year 2020 = Fall 2020 = Winter 2020 


*Total is the average across all regions for all completed TF ls 


8. Average Tier 1 TFI scores by region—Cohort 16 (Fall & Winter 2020) 


plas 44% 41% 
ane 29% 
10) 0 
| ri ; i il PL 
Metro North South Total average 


across all regions 
Fall 2020 Winter 2020 


*Total is the average across all regions for all completed TFls 
Behavioral data system 


School teams were asked to identify which ODR data system(s) they use to track data. Schools are expected to have an 
established behavior system in place by winter training in their first year. Some key findings include: 


» Most schools from the Metro and South regions in Cohort 15 report that they have a behavioral data system in 
place. For most schools from the North region (89%), it is unknown whether they have a data system in place 
(Figure 9). 


» The majority of schools from the Metro and North regions in Cohort 16 have a data system in place. Half (43%) 
of school teams from the South region either do not have a data system or it is unknown what type of data 
system they use to track ODR data. 


» The high number of school teams with an unknown data system could be attributed to a few factors. First, there 
was a low response rate for the winter training survey, which is when school teams are asked which data 
system they are using. Second, school teams may not have updated this information when asked to do so in 
the Wilder Research database. Third, school teams have indicated on the Winter 2020 training survey in 
several of the open-ended questions they are overwhelmed due to the ongoing COVID-19 pandemic and 
therefore are perhaps not as engaged with their PBIS initiatives (or tracking data). 
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9. Schools with behavioral data system, Cohorts 15 & 16 


Cohort 15 
total (N=66) MEO 0 
v4) a ee 
(N=38) 
vate) SS, ST 


= SWIS/SWIS+1 =Infinite Campus =Skyward = Other datasystem = Unknown 


Cohort 16 
total (N=52) ELEC 25700 Cy IS EA 
(x14) CS 7 
ab 25% | 13% | 25% 
(x-22) ES iso ES. 


aSWIS =Infinite Campus =Skyward =Otherdatasystem =Unknown ~© None 


Note. Numbers may not equal 100% due to rounding. 


The presence of a behavioral data system allows for schools to track office discipline referral (ODR) data and 
use this for data-based decision-making. ODR data includes whether the problem behavior, locations, possible 
motivations, others involved, and administrative consequences. The data collected in these systems are used for 
improving school-wide behavior support. 
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Challenges for sharing key data with school staff 


School teams were asked about the challenges associated with sharing PBIS implementation fidelity with other school 
staff. School teams in Cohort 15 and Cohort 16 said they had difficulty sharing this data due to lack of time and 
pandemic-related reasons: 


We just haven't had time - we've been dealing with other more pressing concerns. 


There are no challenges in presenting the data, the challenge comes in regards to there being enough time to fully 
cover the data with all of the other information that needs to be covered in meetings with the ever-changing learning 
models. 


The uncertainty of this year was the biggest hurdle; under normal circumstances, we didn't have any issues. 


We have not shared TFI data with other school staff: At this time, most of our whole staff meetings have addressed 
issues surrounding our learning modules and updates to COVID related processes. 


We have not shared this data because we have not be able to meet school-wide (in person) to discuss the data. 
Additional knowledge and skills needed to implement PBIS 


School teams from both cohorts were invited to provide open-ended, general comments about PBIS training and 
additional knowledge and skills needed to help with PBIS implementation. Four school teams in Cohort 15 said they 
would like more suggestions regarding PBIS and distance or hybrid learning. Three school teams in Cohort 15 
indicated ongoing difficulties with PBIS implementation due to the pandemic. Three other school teams mentioned 
they need more time. Below are some comments, edited for clarity: 


We would like more ideas for implementing PBIS virtually and creative ways to implement positive ways to reinforce 
students and school-wide celebrations. 


It’s still very difficult to plan [for our PBIS initiative] due to uncertainty of the future - distance learning? Hybrid? In- 
Person? 


We need to get together as a team and discuss our goals and carry out our plan. 


I believe this [asynchronous] format would have been incredibly helpful in a typical year (when we could meet in 
person as a team), however the challenges this year was the separation, and the juggling of the other responsibilities 
related to distance learning and navigating schedules. 


We'd like more time to work with our team and see what an informal TFI walk-through for our school would look like. 


When asked about specific knowledge or skills that they need to implement PBIS in their schools, five school teams in 
Cohort 16 indicated they wanted more help with data. Four school teams mentioned they needed more time. Two 
said they would like continued external support and training. Below are some comments, edited for clarity: 


We would like a third year of cohort support. 
At this time, we need more time to work on our action planning goals. 


We would like any examples, ideas of data gathering and a quicker way of do so, and in addition to models that exist to 
help with the action planning goals at our school would be helpful. 
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Issues to consider 


The PBIS State Leadership Team (SLT), the Minnesota Department of Education (MDE), and the Regional 
Implementation Projects (RIPs), should consider the following in order to improve training survey participation, 
usefulness, and outcomes for schools participating in the two-year cohort training sequence. 


Consider an alternative way to 
track participation for school 
teams who participate in the 
asynchronous model. 


Follow up with schools that do 
not complete the training 
surveys and consider asking 
one team representative to 
complete the survey. 


Consider extra support for 
school teams that began their 
first year of training during the 
pandemic. 


Continue to meet schools 
where they are at regarding 
their PBIS initiatives and the 
ongoing COVID-19 pandemic. 


Ina typical year, school team and administrator training attendance is tracked. 
However, given the changes in training delivery formats, attendance data was 
difficult to track (especially for asynchronous training). If the asynchronous model is 
used for future trainings, the SLT and RIPs should consider other methods for 
measuring participation for school teams that use this format. These could include 
completing certain tasks or benchmarks within a specific time frame and meeting 
with an external coach to get more support and guidance on each of these tasks (as 
well as to provide accountability to the school teams for completion). 


Training survey participation was low for Winter 2020 training compared with other 
trainings in previous years. Very few school teams completed an asynchronous 
training survey. This could be due to “low bandwidth” from PBIS team members 
given the COVID-19 pandemic as well as the ever-changing circumstances schools 
face with their learning models. Instead of an online survey at the completion of the 
asynchronous training, perhaps external coaches and trainers could ask 3-4 questions 
at the end of their coaching sessions to get feedback and perceptions of training. 


Cohort 16 school teams that began training during the ongoing COVID-19 pandemic 
seemed to have a harder time getting the most out of PBIS training. Over half of 
Cohort 16 school teams that attended either the live or asynchronous trainings rated 
the training overall to be somewhat useful. More Cohort 16 teams only somewhat 
agreed that they were able to understand concepts and learning modules in the 
asynchronous training format. Similarly, a slightly greater number of school teams 
from Cohort 16 somewhat agreed the training content was clearly presented and that 
they are confident in their ability to share what they learned at PBIS training with 
other school staff. RIPs and trainers should consider extra supports for Cohort 16 
school teams to be sure they are getting the most out of training and are fully 
understanding PBIS training concepts. One person from a school team suggested that 
offering another year of PBIS cohort training could be helpful for school teams. 


School teams in both cohorts mentioned that they are overwhelmed, prioritizing other 
things, and struggling in general due to the ongoing pandemic. RIPs have made 
accommodations regarding PBIS training (offering an asynchronous and live online 
formats) and have continued to support schools in a variety of ways throughout the 
pandemic. The SLT and RIPs should continue to think about ways to support schools 
that have gone through PBIS training during the pandemic. School teams suggested 
the need for ongoing coaching, and perhaps a third year of cohort training—MDE and 
RIPs should consider these options given the circumstances of schools that were in 
cohort training during the pandemic. The RIPs and MDE should also consider 
debriefing what worked, and what did not during the pandemic in order to document 
lessons learned for future use. 
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